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TITLE OF THE INVENTION 
IMAGE SENSING APPARATUS AND ITS CONTROL METHOD, CONTROL 
PROGRAM, AND STORAGE MEDIUM 

FIELD OF THE INVENTION 

The present invention relates to a compact image 
sensing apparatus such as a digital still camera, video 
movie camera, and the like that adopt a solid-state 
image sensing element, its control method, a control 
program, and a storage medium. 

BACKGROUND OF THE INVENTION 

In a digital color camera, a solid-state image 
sensing element such as a CCD, CMOS sensor, or the like 
is exposed to an object image for a desired period of 
time in synchronism with pressing of a release button, 
and an image signal for one frame obtained by exposure 
undergoes a predetermined process such as a YC process 
or the like to obtain an image signal of a 
predetermined format. Digital image signals that 
represent sensed images are recorded in a semiconductor 
memory for respective images. The recorded image 
signals are solely or seguentially read out as needed 
to reproduce displayable or printable signals, which 
are output to and displayed on a monitor or the like. 



The present applicant proposed a technique for 
generating R, G, and B images using a three- or 
four-lens optical system, and compositing these images 
to obtain a video signal. This technique is very 
effective for implementing a low-profile image sensing 
system. 

However, in the method using the three- or 
four-lens optical system, R, G, and B images suffer 
misregistration resulting from the disparity of the 
optical system. 

Misregistration of R, G, and B images results in 
unwanted color misregistration and low sharpness of an 
output image. 

Techniques for obtaining a composite image by 
executing position correction based on a pair of images 
with disparity are known in Japanese Patent Laid-Open 
Nos. 6-6680 and 8-116490, but they are not directed to 
independently obtaining R, G, and B images. 

SUMMARY OF THE INVENTION 
It is an object of the present invention to 

provide an image sensing apparatus which can composite 

images by satisfactorily correcting their 

misregistration and its control method, a control 

program, and a storage medium. 

In order to solve the aforementioned problems and 

to achieve the above object, an image sensing apparatus 



according to the first aspect of the present invention 
is characterized by the following arrangement. 

That is, an image sensing apparatus comprises a 
plurality of apertures that receives external light 
from different positions, a plurality of image sensing 
units that outputs image signals obtained by 
independently receiving light that comes from an 
identical position of an object and is received via the 
plurality of apertures, and independently extracting 
predetermined color components for each received light, 
and a signal processing device that forms a signal that 
outputs an object image by mixing the image signals 
output from the plurality of image sensing units, 
wherein upon forming the signal that outputs the object 
image, the signal processing device determines a 
position deviation state, which occurs between the 
image signals, and corrects the position deviation 
state by a signal process, and forms the signal that 
outputs the object image. 

An image sensing apparatus according to the 
second aspect of the present invention is characterized 
by the following arrangement. 

That is, an image sensing apparatus comprises an 
image sensing element having a plurality of image 
sensing areas, a photographing optical system that 
forms object images on the plurality of image sensing 
areas via a plurality of imaging systems corresponding 



to the plurality of image sensing areas, a temperature 
measuring device that detects a temperature near the 
image sensing element, and a signal processing device 
that forms a signal that outputs an object image by 
mixing image signals output from the plurality of image 
sensing areas of the image sensing element, wherein the 
signal processing device corrects a position deviation 
between the image signals in accordance with an output 
from the temperature measuring device upon forming the 
signal that outputs the object image, and forms the 
signal that outputs the object image. 

An image sensing apparatus according to the third 
aspect of the present invention is characterized by the 
following arrangement. 

That is, an image sensing apparatus comprises an 
image sensing element having a plurality of image 
sensing areas, a photographing optical system that 
forms object images on the plurality of image sensing 
areas via a plurality of imaging systems corresponding 
to the plurality of image sensing areas, and a signal 
processing device that forms a signal that outputs an 
object image by mixing image signals output from the 
plurality of image sensing areas of the image sensing 
element, wherein the signal processing device corrects 
a position deviation between the image signals in 
accordance with an object distance upon forming the 



signal that outputs the object image, and forms the 
signal that outputs the object image. 

An image sensing apparatus according to the 
fourth aspect of the present invention is characterized 
5 by the following arrangement. 

That is, an image sensing apparatus comprises an 
image sensing element having a plurality of image 
sensing areas, a photographing optical system that 
forms object images on the plurality of image sensing 
10 areas via a plurality of imaging systems corresponding 
to the plurality of image sensing areas, and a signal 
processing device that forms a signal that outputs an 
object image by mixing image signals output from the 
plurality of image sensing areas of the image sensing 
15 element, wherein the signal processing device corrects 
a position deviation between the image signals in 
response to setting of a near-distance photographing 
mode upon forming the signal that outputs the object 
image, and forms the signal that outputs the object 
20 image. 

An image sensing apparatus according to the fifth 
aspect of the present invention is characterized by the 
following arrangement. 

That is, an image sensing apparatus comprises an 
25 image sensing element having first and second image 
sensing areas with substantially the same size on a 
single plane, a photographing optical system that 
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respectively forms first and second object images on 
the first and second image sensing areas, and a signal 
processing device that processes an output signal from 
the image sensing element, and each of the first and 
second image sensing areas has a matrix of a plurality 
of pixels arranged at a pitch a in the horizontal 
direction and a pitch b in the vertical direction on a 
light-receiving surface, the first and second image 
sensing areas have a positional relationship in which 
the first and second image sensing areas are separated 
a x h x c in the horizontal direction and b x c in the 
vertical direction (where h is a positive integer) , the 
image sensing element forms first and second images 
which are formed to have an identical spectral 
distribution and have substantially the same fields of 
view, and the signal processing device generates a 
composite image signal based on the first and second 
images . 

A control method of an image sensing apparatus 
according to the first aspect of the present invention 
is characterized by the following arrangement. 

That is, a method of controlling an image sensing 
apparatus, which comprises a plurality of apertures for 
receiving external light from different positions, and 
a plurality of image sensing units for outputting image 
signals obtained by independently receiving light that 
comes from an identical position of an object and is 



received via the plurality of apertures, and 
independently extracting predetermined color components 
for each received light, comprises the step of forming 
a signal that outputs an object image by mixing the 
image signals output from the plurality of image 
sensing units, and correcting, when it is determined 
that a predetermined position deviation occurs between 
the image signals upon forming the signal that outputs 
the object image, the predetermined position deviation 
by a signal process and forming the signal that outputs 
the object image. 

A control method of an image sensing apparatus 
according to the second aspect of the present invention 
is characterized by the following arrangement. 

That is, a method of controlling an image sensing 
apparatus, which comprises an image sensing element 
having a plurality of image sensing areas, a 
photographing optical system for forming object images 
on the plurality of image sensing areas via a plurality 
of imaging systems corresponding to the plurality of 
image sensing areas, and a temperature measuring device 
for detecting a temperature near the image sensing 
element, comprises the step of forming a signal that 
outputs an object image by mixing image signals output 
from the plurality of image sensing areas of the image 
sensing element, and correcting a position deviation 
between the image signals in accordance with an output 



from the temperature measuring device upon forming the 
signal that outputs the object image and forming the 
signal that outputs the object image. 

A control method of an image sensing apparatus 
according to the third aspect of the present invention 
is characterized by the following arrangement. 

That is, a method of controlling an image sensing 
apparatus, which comprises an image sensing element 
having a plurality of image sensing areas, and a 
photographing optical system for forming object images 
on the plurality of image sensing areas via a plurality 
of imaging systems corresponding to the plurality of 
image sensing areas, comprises the step of forming a 
signal that outputs an object image by mixing image 
signals output from the plurality of image sensing 
areas of the image sensing element, and correcting a 
position deviation between the image signals in 
accordance with an object distance upon forming the 
signal that outputs the object image, and forming the 
signal that outputs the object image. 

A control method of an image sensing apparatus 
according to the fourth aspect of the present invention 
is characterized by the following arrangement. 

That is, a method of controlling an image sensing 
apparatus, which comprises an image sensing element 
having a plurality of image sensing areas, and a 
photographing optical system for forming object images 



on the plurality of image sensing areas via a plurality 
of imaging systems corresponding to the plurality of 
image sensing areas, comprises the step of forming a 
signal that outputs an object image by mixing image 
signals output from the plurality of image sensing 
areas of the image sensing element, and correcting a 
position deviation between the image signals in 
response to setting of a near-distance photographing 
mode upon forming the signal that outputs the object 
image, and forming the signal that outputs the object 
image . 

A control program according to the first aspect 
of the present invention is characterized by the 
following arrangement. 

That is, a control program for controlling an 
image sensing apparatus, which comprises a plurality of 
apertures for receiving external light from different 
positions, and a plurality of image sensing units for 
outputting image signals obtained by independently 
receiving light that comes from an identical position 
of an object and is received via the plurality of 
apertures, and independently extracting predetermined 
color components for each received light, comprises a 
code of the step of forming a signal that outputs an 
object image by mixing the image signals output from 
the plurality of image sensing units, and correcting, 
when it is determined that a predetermined position 



deviation occurs between the image signals upon forming 
the signal that outputs the object image, the 
predetermined position deviation by a signal process, 
and forming the signal that outputs the object image. 
5 A control program according to the second aspect 

of the present invention is characterized by the 
following arrangement. 

That is, a control program for controlling an 
image sensing apparatus, which comprises an image 
10 sensing element having a plurality of image sensing 
areas, a photographing optical system for forming 
object images on the plurality of image sensing areas 
via a plurality of imaging systems corresponding to the 
plurality of image sensing areas, and a temperature 
15 measuring device for detecting a temperature near the 
image sensing element, comprises a code of the step of 
forming a signal that outputs an object image by mixing 
image signals output from the plurality of image 
sensing areas of the image sensing element, and 
20 correcting a position deviation between the image 
signals in accordance with an output from the 
temperature measuring device upon forming the signal 
that outputs the object image, and forming the signal 
that outputs the object image. 
25 A control program according to the third aspect 

of the present invention is characterized by the 
following arrangement. 
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That is, a control program for controlling an 
image sensing apparatus, which comprises an image 
sensing element having a plurality of image sensing 
areas, and a photographing optical system for forming 
object images on the plurality of image sensing areas 
via a plurality of imaging systems corresponding to the 
plurality of image sensing areas, comprises a code of 
the step of forming a signal that outputs an object 
image by mixing image signals output from the plurality 
of image sensing areas of the image sensing element, 
and correcting a position deviation between the image 
signals in accordance with an object distance upon 
forming the signal that outputs the object image, and 
forming the signal that outputs the object image. 

A control program according to the fourth aspect 
of the present invention is characterized by the 
following arrangement. 

That is, a control program for controlling an 
image sensing apparatus, which comprises an image 
sensing element having a plurality of image sensing 
areas, and a photographing optical system for forming 
object images on the plurality of image sensing areas 
via a plurality of imaging systems corresponding to the 
plurality of image sensing areas, comprises a code of 
the step of forming a signal that outputs an object 
image by mixing image signals output from the plurality 
of image sensing areas of the image sensing element, 



and correcting a position deviation between the image 
signals in response to setting of a near-distance 
photographing mode upon forming the signal that outputs 
the object image, and forming the signal that outputs 
the object image. 

A storage medium according to the present 
invention is characterized by the following arrangement. 

That is, the control program is computer-readably 
stored. 

Other objects and advantages besides those 
discussed above shall be apparent to those skilled in 
the art from the description of a preferred embodiment 
of the invention which follows. In the description, 
reference is made to accompanying drawings, which form 
a part hereof, and which illustrate an example of the 
invention. Such example, however, is not exhaustive of 
the various embodiments of the invention, and therefore 
reference is made to the claims which follow the 
description for determining the scope of the invention. 

BRIEF DESCRIPTION OF THE DRAWINGS 
Fig. 1 is a sectional view of a digital color 
camera; 

Fig. 2 is a rear view of the digital color 
camera; 

Fig. 3 is a side view of the camera viewed from 
the left side in Fig. 2; 
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Fig. 4 is a side view of the camera viewed from 
the right side in Fig. 2; 

Fig. 5 is a detailed view of an image sensing 
system; 

Fig. 6 is a plan view of a stop; 

Fig. 7 shows a photographing lens viewed from the 
light exit side; 

Fig. 8 is a front view of a solid-state image 
sensing element; 

Fig. 9 shows the photographing lens viewed from 
the light incidence side; 

Fig. 10 is a graph showing the spectral 
transmission characteristics of an optical filter; 

Fig. 11 is a view for explaining the function of 
microlenses; 

Fig. 12 is a sectional view of a lens portion; 

Fig. 13 is a view for explaining the spacing 
setup process of lens portions of the photographing 
lens; 

Fig. 14 is a view for explaining the positional 
relationship between an object image and image sensing 
areas; 

Fig. 15 is a view for explaining the positional 
relationship of pixels upon projecting the image 
sensing areas onto an object; 

Fig. 16 is a perspective view of first and second 
prisms which form a viewfinder; 
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Fig. 17 is a sectional view of a viewfinder 
system; 

Fig. 18 is a block diagram of a signal processing 
system; 

Fig. 19 is a view for explaining a state wherein 
the positions of elements of the image sensing system 
have changed due to thermal expansion; 

Fig. 20 is a view for explaining the addresses of 
image signals from the image sensing areas; 

Fig. 21 is a block diagram showing a signal 
processing system which uses the output from a distance 
measurement device in correction of misregistration; 

Fig. 22 is a view showing the optical path when 
an actual object distance increases to infinity; 

Fig. 23 is a view for explaining pixels on the 
solid-state image sensing element used to detect 
misregistration; 

Fig. 24 is a graph showing signals of pixel 
sequences of two image sensing areas; 

Fig. 25 is a graph showing signals of pixel 
sequences of two image sensing areas; 

Fig. 26 is a view for explaining pixels on the 
solid-state image sensing element used to detect 
misregistration; and 

Fig. 27 is a view for explaining pixels on the 
solid-state image sensing element used to detect 
misregistration. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 
Preferred embodiments of the present invention 
will be described in detail hereinafter with reference 
to the accompanying drawings. 
(First Embodiment) 

Figs. 2, 3, and 4 show the outer appearance of a 
digital color camera according to the first embodiment 
of the present invention, in which Fig. 2 is a rear 
view of the camera, Fig. 3 is a side view of the camera 
viewed from the left side in Fig. 2, and Fig. 4 is a 
side view of the camera viewed from the right side in 
Fig. 2. 

Referring to Figs. 2, 3, and 4, reference numeral 
101 denotes a card-shaped camera main body; 105, a main 
switch; 106, a release button (also shown in Fig. 1) ; 
107, 108, and 109, switches with which the user sets 
the camera state; and 150, an indicator of the 
remaining photographable count. Reference numeral 111 
denotes a viewfinder eyepiece window, from which object 
light that has entered the viewfinder exits. Reference 
numeral 114 denotes a standardized connection terminal 
which is connected to an external computer or the like 
to exchange data; 200, a contact protection cap which 
also serves as a grip; 120, a projection which is 
formed coaxial with the release button 106 arranged on 
the front surface of the camera; and 890, an image 



sensing system which is incorporated in the camera. 
The contact protection cap 200 is formed of a soft 
resin or rubber. 

The camera main body 101 may have the same size 
as a PC card and may be inserted into a personal 
computer. In this case, the camera main body 101 has a 
length of 85.6 mm, a width of 54.0 mm, and a thickness 
of 3.3 mm (PC card standard Type 1) or a thickness of 
5.0 mm (PC card standard Type 2). 

Fig. 1 is a sectional view of the digital color 
camera taken along a plane that passes through the 
release button 106, image sensing system 890, and 
viewfinder eyepiece window 111. Referring to Fig. 1, 
reference numeral 123 denotes a housing for holding the 
respective building components of the camera; 125, a 
back cover; 890, an image sensing system; 121, a switch 
which is turned on upon pressing of the release button 
106; and 124, a coil spring that biases the release 
button 106 in a projecting direction. The switch 121 
comprises a first-stage circuit which is closed when 
the release button 106 is pressed to its half-stroke 
position, and a second-stroke circuit which is closed 
when the release button 106 is pressed to its 
full-stroke position. 

Reference numerals 112 and 113 denote first and 
second prisms which form a viewfinder optical system. 
The first and second prisms 112 and 113 are formed of a 



transparent material such as an acrylic resin or the 
like, and have an identical refractive index. Also, 
these prisms 112 and 113 are solid so that light rays 
propagate straight through their interiors. A 
light-shielding printed region 113b is formed on the 
peripheral portion of an object light exit surface 113a 
of the second prism 113, thus limiting the passing 
range of viewfinder exit light. This printed region 
also extends to the side surfaces of the second prism 
113 and to portions facing the object light exit 
surface 113a. 

The image sensing system 8 90 is constructed by 
attaching a protection glass 160, photographing lens 
800, sensor board 161, and relay members 163 and 164 
used to adjust the sensor position to a housing 123. 
On the sensor board 161, a solid-state image sensing 
element 820, sensor cover glass 162, and temperature 
sensor 165 as temperature measurement means are 
attached, and a stop 810 (to be described later) is 
adhered to the photographing lens 800. After the relay 
members 163 and 164 are movably fitted into through 
holes 123a and 123b of the housing and the 
photographing lens 800 and the solid-state image 
sensing element 820 are adjusted to have an appropriate 
positional relationship, these members are adhered to 
the sensor board 161 and housing 123. 



Furthermore, the protection glass 160 and sensor 
cover glass 162 have light-shielding printed regions 
except for an effective portion so as to reduce light, 
which comes from a range outside the image sensing 
range and enters the solid-state image sensing element 
820, as much as possible. In Fig. 1, reference 
numerals 162a and 162b denote printed regions. Also, 
these glasses have an antiref lection coat except for 
the printed regions to avoid ghost from being produced. 

Details of the image sensing system 890 will be 
described below. 

Fig. 5 is a detailed view of the image sensing 
system 890. Basic elements of a photographing optical 
system are the photographing lens 800, stop 810, and 
solid-state image sensing element 820. The image 
sensing system 890 comprises four optical systems for 
independently obtaining a green (G) image signal, red 
(R) image signal, and blue (B) image signal. 

Since the assumed object distance is as very 
large as several m compared to the optical path length 
of an imaging system, if an incidence surface is 
aplanatic with respect to the assumed object distance, 
the incidence surface is a concave surface having a 
very small curvature, and is replaced by a plane in 
this embodiment. 

As shown in Fig. 7 viewed from the light exit 
side, the photographing lens 800 has four lens portions 



800a, 800b, 800c, and 800d, each of which is formed by 
a zone spherical surface. Infrared ray cut filters 
having a low transmittance with respect to a wavelength 
range not less than 670 nm are formed on these lens 
portions 800a, 800b, 800c, and 800d, and a 
light-shielding film is formed on a flat portion 800f 
indicated by hatching. 

The four lens portions 800a, 800b, 800c, and 800d 
respectively form imaging systems. As will be 
described later, the lens portions 800a and 800d are 
used to sense a green (G) image signal, the lens 
portion 800b is used to sense a red (R) image signal, 
and the lens portion 800c is used to sense a blue (B) 
image signal. All focal lengths at R, G, and B 
representative wavelengths are 1.45 mm. 

In order to suppress high-frequency components of 
an object image equal to or higher than the Nyquist 
frequency determined by the pixel pitch of the 
solid-state image sensing element 820, and to improve 
response on the low-frequency side, a light incidence 
surface 800e of the photographing lens 800 has 
transmittance distribution regions 854a and 854b. This 
is called an apodization, which is a scheme for 
obtaining a desired MTF by providing characteristics 
that set a highest transmittance at the center of the 
stop, and lower the transmittance toward the outer 
periphery. 



The stop 810 has four circular apertures 810a, 
810b, 810c, and 810d, as shown in Fig. 6. Object light 
components which have entered the light incidence 
surface 800e via these apertures exit from the four 
lens portions 800a, 800b, 800c, and 800d, and form four 
objective images on the image sensing surface of the 
solid-state image sensing element 820. The stop 810, 
the light incidence surface 800e, and the image sensing 
surface of the solid-state image sensing element 820 
are arranged parallel to each other. The stop 810 and 
four lens portions 800a, 800b, 800c, and 800d are set 
to have a positional relationship that satisfies the 
Twynken-Sommer condition, i.e., that which can 
simultaneously remove coma and astigmatism. 

By dividing the lens portions 800a, 800b, 800c, 
and 800d into zone patterns, the curvature of field can 
be satisfactorily corrected. An image surface formed 
by a single spherical surface is that expressed by the 
Petzval's curvature, but the image surface is 
planarized by joining a plurality of such spherical 
surfaces. As shown in Fig. 12 as a sectional view of 
the lens portion, all the spherical surfaces of the 
zones have an identical central position PA to satisfy 
a condition for removing coma and astigmatism, and when 
the lens portions 800a, 800b, 800c, and 800d are 
divided in such patterns, the curvatures of an object 
image produced by the respective zones become perfectly 



equal to each other, thus obtaining high overall MTF 
characteristics. The remaining curvatures in this case 
can be corrected by an arithmetic process. If the 
respective lens portions produce identical curvatures, 
a correction process can be simplified. 

The radii of the zone spherical surfaces are set 
to increase like arithmetic series from the central 
zone toward the outer periphery, and the increment is 
given by mX/(n-l) (where X is the representative 
wavelength of an image formed by each lens portion, n 
is the refractive index of the photographing lens 800 
with respect to this representative wavelength, and m 
is a positive constant) . With this arrangement, the 
optical path length difference of light rays that pass 
neighboring zones is mk, and exit light rays are in 
phase. When each lens portion is divided into a larger 
number of zones, it serves as a diffraction optical 
element . 

In order to minimize flare produced at step 
portions of the zones, steps parallel to principal rays 
are formed, as shown in Fig. 12. Since the lens 
portions 800a, 800b, 800c, and 800d are away from a 
pupil, such arrangement can provide a great flare 
suppression effect. 

Fig. 8 is a front view of the solid-state image 
sensing element 820. The solid-state image sensing 
element 820 has four image sensing areas 820a, 820b, 
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820c, and 820d in correspondence with four object 
images to be formed. Although simply illustrated in 
Fig. 8, each of the image sensing areas 820a, 820b, 
820c, and 820d is a 1,248 mm x 0.936 mm area where 800 
X 600 pixels are arranged to have vertical and 
horizontal pitches P of 1.56 |im, and the diagonal size 
of each image sensing area is 1.56 mm. A 0.156 mm 
(horizontal) x 0.468 mm (vertical) separation band is 
formed between neighboring image sensing areas. 
Therefore, the distances between the centers of the 
image sensing areas are the same in the horizontal and 
vertical directions, i.e., 1.404 mm. 

That is, if the image sensing areas 820a and 820d 
have horizontal pitch a = P, vertical pitch b - P, 
constant c = 900, and positive integer h = 1 on the 
light-receiving surface, these areas have a positional 
relationship in which they are separated a X h X c in 
the horizontal direction and b x c in the vertical 
direction within the light-receiving surface. By 
forming such relationship, misregistration caused by a 
change in temperature and a change in object distance 
can be corrected by a very simple arithmetic operation. 
Note that misregistration is mismatch of object image 
sampling positions caused among image sensing systems 
(e.g., R image sensing system/G image sensing system/B 
image sensing system) having different light-receiving 
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spectrum distributions in, e.g., a multi-sensor color 
camera or the like. 

Reference numerals 851a, 851b, 851c, and 851d in 
Fig. 8 denote image circles within which object images 
are formed. The maximum shape of each of the image 
circles 851a, 851b, 851c, and 851d is a circular shape 
determined by the sizes of the aperture of the stop and 
the exit-side spherical surface portion of the 
photographing lens 800 although their outer peripheral 
portions suffer illuminance drop due to the effect of 
the printed regions 162a and 162b formed on the 
protection glass 160 and sensor cover glass 162. 
Therefore, the image circles 851a, 851b, 851c, and 851d 
have overlapping portions. 

Referring back to Fig. 5, hatched and 
cross-hatched portions 852a and 852b in a region 
sandwiched between the stop 810 and photographing lens 
800 are optical filters, which are formed on the light 
incidence surface 800e of the photographing lens 800. 
As shown in Fig. 9 that shows the photographing lens 
800 viewed from the light incidence side, the optical 
filters 852a, 852b, 852c, and 852d are formed on ranges 
that completely include the stop apertures 810a, 810b, 
810c, and 810d. 

The optical filters 852a and 852d have spectral 
transmission characteristics that mainly transmit green, 
as indicated by G in Fig. 10. The optical filter 852b 



has spectral transmission characteristics that mainly 
transmit red, as indicated by R. Furthermore, the 
optical filter 852c has spectral transmission 
characteristics that mainly transmit blue, as indicated 
by B. That is, these filters are primary color filters. 
As the products with the characteristics of the 
infrared ray cut filters formed on the lens portions 
800a, 800b, 800c, and 800d, object images are formed on 
the image circles 851a and 851d, 851b, and 851c by 
green, red, and blue light components, respectively. 

When nearly equal focal lengths for the 
representative wavelengths of the spectrum 
distributions are set in the respective imaging systems, 
a color image that has satisfactorily undergone 
chromatic aberration correction can be obtained by 
compositing these image signals. Normally, an 
achromatic process for removing chromatic aberration 
requires a combination of at least two lenses having 
different dispersions. By contrast, since each imaging 
system has one lens, a cost reduction effect is 
expected. Furthermore, a low-profile image sensing 
system can be easily realized. 

On the other hand, optical filters are formed on 
the four image sensing areas 820a, 820b, 820c, and 820d 
of the solid-state image sensing element 820. The 
spectral transmission characteristics of the image 
sensing areas 820a and 820d, 820b, and 820c are 



respectively those indicated by G, R, and B in Fig. 10. 
That is, the image sensing areas 820a and 820d, 820b, 
and 820c respectively have sensitivities to green (G) , 
red (R) , and blue (B) light components. 

Since the light-receiving spectrum distribution 
of each image sensing area is given by the product of 
spectral transmittances of the pupil and image sensing 
area, even when image circles overlap each other, a 
combination of the pupil and image sensing area of the 
imaging system is nearly selected by the wavelength 
range. 

Furthermore, microlenses 821 are formed on the 
image sensing areas 820a, 820b, 820c, and 820d for 
respective light-receiving portions (e.g., 822a, 822b) 
of pixels. Each microlens 821 is decentered with 
respect to the corresponding light-receiving portion of 
the solid-state image sensing element 820, and the 
decentering amount is set at zero at the center of each 
of the image sensing areas 820a, 820b, 820c, and 820d 
and becomes larger toward the outer peripheral portion. 
The decentering direction is a direction of a line 
segment that connects the center of each of the image 
sensing areas 820a, 820b, 820c, and 820d and each 
light-receiving portion. 

Fig. 11 is a view for explaining the effect of 
the microlenses, and is an enlarged sectional view of 
neighboring light-receiving portions 822a and 822b of 



the image sensing areas 820a and 820b. A microlens 
821a is decentered upward in Fig. 11 with respect to 
the light-receiving portion 822a, while the microlens 
821b is decentered downward in Fig. 11 with respect to 
5 the light-receiving portion 822b. As a result, a light 
beam that enters the light-receiving portion 822a is 
limited to a hatched region 823a, and a light beam that 
enters the light-receiving portion 822b is limited to a 
hatched region 823b. 
10 The light beam regions 823a and 823b have tilts 

in opposite directions, and are directed toward the 
lens portions 800a and 800b. Therefore, by 
appropriately selecting the decentering amounts of the 
microlenses, only light beams that have left specific 
15 pupils enter the respective image sensing areas. That 
is, the decentering amounts can be set so that object 
light components that respectively have passed through 
the apertures 810a, 810b, 810c, and 810d of the stop 
are photoelectrically converted mainly by the image 
20 sensing areas 820a, 820b, 820c, and 820d, respectively. 

In addition to the aforementioned scheme for 
selectively assigning the pupils to the respective 
image sensing areas using the wavelength ranges, a 
scheme for selectively assigning the pupils to the 
25 respective image sensing areas using the microlenses is 
applied, and the printed regions are formed on the 
protection glass 160 and sensor cover glass 162. In 
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this way, crosstalk among the wavelengths can be 
reliably prevented while allowing overlaps of the image 
circles. That is, object light components that 
respectively have passed through the apertures 810a, 
810b, 810c, and 810d of the stop are photoelectrically 
converted by the image sensing areas 820a, 820b, 820c, 
and 820d, respectively. Therefore, the image sensing 
areas 820a and 820d, 820b, and 820c respectively output 
G, R, and B signals. 

An image processing system (not shown) forms a 
color image on the basis of the selective photoelectric 
conversion outputs that the plurality of image sensing 
areas of the solid-state image sensing element 820 
obtain from one of a plurality of object images. In 
this case, a signal process for correcting distortions 
of the imaging systems by arithmetic operations, and 
forming a color image with reference to a G image 
signal that contains the peak wavelength of 555 nm of 
the spectral luminous efficiency is executed. Since a 
G object image is formed on the two image sensing areas 
820a and 820b, the number of pixels of the G object 
image is twice that of an R or B image signal, and an 
image which has high resolution especially in the 
wavelength range with high visual sensitivity can be 
obtained. In this case, a so-called pixel shift scheme 
for increasing the resolution by a fewer number of 
pixels by shifting object images on the image sensing 



areas 820a and 820d of the solid-state image sensing 
element 1/2 pixels in the vertical and horizontal 
directions is used. As shown in Fig. 8, object image 
centers 860a, 860b, 860c, and 860d which are also the 
centers of the image circles are respectively offset 
1/4 pixels from the centers of the image sensing areas 
820a, 820b, 820c, and 820d in the directions of arrows 
861a, 861b, 861c, and 861d, thus implementing a 1/2 
pixel shift as a whole. Note that the lengths of the 
arrows 861a, 861b, 861c, and 861d do not represent the 
offset amount in Fig. 8. 

Compared to an imaging system using a single 
photographing lens, if the pixel pitch of the 
solid-state image sensing element is fixed, the size of 
the object image of the scheme of this embodiment 
becomes 1/V4 that of a Bayer matrix scheme in which R, 
G, and B color filters are formed on a solid-state 
image sensing element for respective sets of 2 x 2 
pixels. Accordingly, the focal length of the 
photographing lens is shortened approximately to 1/V4 = 
1/2. Therefore, the present invention is very 
effective to realize a low-profile camera. 

The positional relationship between the 
photographing lens and image sensing areas will be 
explained below. As described above, each image 
sensing area has a size of 1.248 mm x 0.936 mm, and 
these image sensing areas neighbor in the horizontal 



direction via a 0.156 mm (horizontal) x 0.468 mm 
(vertical) separation band. The spacing between the 
centers of the neighboring image sensing areas is 1.404 
mm in the vertical and horizontal directions, and is 
1.9856 mm in the diagonal direction. 

Paying attention to the image sensing areas 820a 
and 820d, assume that an image of an object located at 
a position of a reference object distance of 2.38 m is 
formed on the image sensing portion at intervals of 
1.9845 mm obtained by subtracting the diagonal size for 
a 0.5 pixel from 1.9856 mm as the image sensing area 
spacing so as to implement a pixel shift. As a result, 
the spacing between the lens portions 800a and 800d of 
the photographing lens 800 is set at 1.9832 mm, as 
shown in Fig. 13. In Fig. 13, arrows 855a and 855d are 
symbols that represent imaging systems having a 
positive power by the lens portions 800a and 800d of 
the photographing lens 800, rectangles 856a and 856d 
are symbols that represent the ranges of the image 
sensing areas 820a and 820d, and L801 and L802 indicate 
the optical axes of the imaging systems 855a and 855d. 
Since the light incidence surface 800e of the 
photographing lens 800 is a plane, and the lens 
portions 800a and 800d as light exit surfaces are 
Fresnel lenses defined by concentric spherical surfaces, 
a straight line which is perpendicular to the light 



incidence surface via the center of a sphere is an 
optical axis. 

The positional relationship between the object 
images and image sensing areas, and that of pixels upon 
projection onto an object will be explained below while 
reducing the numbers of pixels in the vertical and 
horizontal directions to 1/100 for the sake of 
simplicity. Figs. 14 and 15 are explanatory views of 
them. 

Referring first to Fig. 14, reference numerals 
320a, 320b, 320c, and 320d denote four image sensing 
areas of the solid-state image sensing element 820. 
For the sake of simplicity, each of the image sensing 
areas 320a, 320b, 320c, and 320d has an 8 X 6 pixel 
matrix. The image sensing areas 320a and 320d, 320b, 
and 320c respectively output G, R, and B image signals. 
Pixels in the image sensing areas 320a and 320d are 
indicated by open rectangles, those in the image 
sensing area 320b are indicated by hatched rectangles, 
and those in the image sensing area 320c are indicated 
by closed rectangles. 

A separation band having a size of one pixel 
(horizontal) x 3 pixels (vertical) is formed between 
neighboring image sensing areas. Therefore, the 
distances between the centers of the image sensing 
areas for outputting G images are equal to each other 
in the horizontal and vertical directions. 



- 30 - 



Reference numerals 351a, 351b, 351c, and 351d 
denote object images. To attain a pixel shift, centers 
360a, 360b, 360c, and 360d of the object images 351a, 
351b, 351c, and 351d are offset 1/4 pixels from the 
5 centers of the image sensing areas 320a, 320b, 320c, 
and 320d in the direction of a center 320e of the 
entire image sensing area. 

As a result, the image sensing areas are 
projected backward onto a plane at a predetermined 

10 distance on the object field side, as shown in Fig. 15. 
On the object field side as well, a backward projected 
image of the pixels in the image sensing areas 320a and 
320d is indicated by open rectangles 362a, that in the 
image sensing area 320b is indicated by hatched 

15 rectangles 362b, and that in the image sensing area 
320c is indicated by closed rectangles 362c. 

The backward projected images of the centers 360a, 
360b, 360c, and 360d of the object images overlap each 
other as a point 361, and the respective pixels in the 

20 image sensing areas 320a, 320b, 320c, and 320d are 

projected backward so that their centers do not overlap 
each other. Since the open, hatched, and closed 
rectangles respectively output G, R, and B image 
signals, sampling equivalent to an image sensing 

25 element having color filters in a Bayer matrix can be 
done on an object. 
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A viewfinder system will be explained below. The 
low-profile structure of this viewfinder device is 
realized using the nature that light is totally 
reflected by an interface between a medium with a high 
refractive index and that with a low refractive index. 
The arrangement used in the air will be explained. 

Fig. 16 is a perspective view of the first and 
second prisms 112 and 113 which form the viewfinder. 
The first prism 112 has four surfaces 112c, 112d, 112e, 
and 112f at a position facing a surface 112a (see 
Fig. 17), and object light that has entered the prism 
from the surface 112a leaves the prism from the 
surfaces 112c, 112d, 112e, and 112f. The surfaces 112a, 
112c, 112d, 112e, and 112f are planes. 

On the other hand, the second prism 113 has 
surfaces 113c, 113d, 113e, and 113f at positions facing 
the surfaces 112c, 112d, 112e, and 112f of the first 
prism 112. Object light that has entered the prism 
from the surfaces 113c, 113d, 113e, and 113f leaves the 
prism from the surface 113a. The surfaces 112c, 112d, 
112e, and 112f of the first prism 112 oppose the 
surfaces 113c, 113d, 113e, and 113f of the second prism 
113 via a small air gap. Therefore, the surfaces 113c, 
113d, 113e, and 113f of the second prism 113 are also 
planes . 

In order to allow the user to observe an object 
while he or she brings an eye close to the viewfinder, 
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the viewfinder system has no refracting power. 
Therefore, since the object light incidence surface 
112a of the first prism 112 is a plane, the object 
light incidence surface 113a of the second prism 113 is 
also a plane. In addition, these surfaces are parallel 
to each other. Furthermore, since the image sensing 
system 890 and a signal processing system obtain a 
rectangular image as a total process including 
arithmetic distortion correction, an observation field 
that can be observed via the viewfinder must have a 
rectangular shape. Therefore, all the optically 
effective surfaces of the first and second prisms 112 
and 113 are symmetrical to each other in the vertical 
and horizontal directions. A line of intersection 
between two symmetric surfaces is a viewfinder optical 
axis LI. 

Fig. 17 is a view for explaining the role of the 
surfaces which oppose each other via an air gap. 
Fig. 17 is a top view of a state wherein the viewfinder 
system is formed by combining the first and second 
prisms 112 and 113 to have a predetermined positional 
relationship, and an optical path from the eye position 
of the observer is traced backward in the principal 
section . 

Referring to Fig. 17, a point PI is a point 
farthest from the viewfinder, where the entire 
observation field can be overlooked when the pupil of 
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the eye of the observer is stopped down to infinity, 
i.e., a so-called an eye point. 

A light ray 130 which comes from the point PI and 
has an angle slightly exceeding a viewfinder field 
angle co will be examined. The light ray 130 is 
refracted by the surface 130a of the second prism 113, 
and reaches the surface 113c. The inclination angle of 
the surface 113c is set so that an incident angle (3 of 
a light ray corresponding to the viewfinder field angle 
CO becomes a critical angle. Therefore, the incident 
angle of the light ray 130 on the incidence surface 
113c slightly exceeds the critical angle. As a result, 
the light ray 130 cannot emerge from the surface 113c, 
and is totally reflected. The light-shielding printed 
region 113b is formed on each side surface of the 
second prism 113, and the light ray 130 is absorbed 
there. Therefore, the observer cannot observe an 
object in a direction of the light ray 130, and 
recognizes only a dark portion indicating a range 
outside the object field. 

A light ray 131 which comes from the point PI and 
has an angle slightly smaller than the viewfinder field 
angle 0) will be examined. The light ray 131 is 
refracted by the surface 113a of the second prism 113 
and reaches the surface 113c. As described above, the 
inclination angle of the surface 113c is set so that 
the incident angle p of a light ray corresponding to 
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the viewfinder field angle CO becomes a critical angle. 
The incident angle of the light ray 131 on the surface 
113c is slightly smaller than the critical angle. The 
light ray 131 emerges from the surface 113c, passes 
through a small air gap, and then enters the surface 
112c of the first prism 112. Since the surface 113c 
and the surface 112c opposite to it have the same shape, 
the direction in which the light ray travels in the 
first prism 112 is the same as that in the second prism 
113. 

For the light ray that has reached the first 
prism 112, the total characteristics of the first and 
second prisms 112 and 113 are eguivalent to a 
plane-parallel plate. As a result, the light ray 131 
emerges from the surface 112a with an angle egual to 
its incident angle to the surface 113a. That is, the 
field angle P is equal to the viewfinder field angle CO. 
Therefore, the observer can observe an object in the 
direction of the light ray 131, and can recognize an 
object field. The optical paths of the light rays 130 
and 131 reveal that the viewfinder field can be limited 
by exploiting the critical angle, i.e., a clear 
boundary of the viewfinder field can be obtained. 

Since the first and second prisms 112 and 113 
have symmetrical shapes, as described above, optical 
paths symmetrical to those shown in Fig. 17 about the 
viewfinder optical axis LI are present. Furthermore, 



the limitation of the viewfinder field is similarly 
imposed on the relationships between the surfaces 112e 
and 112f of the first prism 112, and the surfaces 113e 
and 113f of the second prism 113. In the above 
examination, the light rays are traced backward from 
the eye position of the observer for the sake of 
simplicity. Upon examining the optical path in a 
direction in which light coming from an object travels, 
since light rays have reversibility, the above 
examinations are equivalent to that object light that 
has entered the object light incidence surface 112a of 
the first prism 112 from a range within the observation 
field passes through the air gap, and object light that 
has entered the object light incidence surface 112a of 
the first prism 112 from a range outside the 
observation field does not pass through the air gap. 
Therefore, as the total viewfinder characteristics, a 
roughly rectangular viewfinder field can be obtained 
from the position of the point PI. 

A schematic arrangement of the signal processing 
system will be explained below. 

Fig. 18 is a block diagram of the signal 
processing system. This camera is a single-sensor 
digital color camera which uses a solid-state image 
sensing element 120 such as a CCD, CMOS sensor, or the 
like, and obtains an image signal that represents a 
moving image or still image by driving the solid-state 



image sensing element 120 continuously or in a one-shot 
manner. Note that the solid-state image sensing 
element 120 is an image sensing device of a type which 
converts exposed light into an electrical signal for 
respective pixels, accumulates a charge corresponding 
to the light amounts, and reads out the charge. 

Note that Fig. 18 illustrates only blocks which 
are directly related to the present invention, and an 
illustration and description of those which are not 
directly related to the present invention will be 
omitted. 

As shown in Fig. 18, the image sensing apparatus 
has an image sensing system 10, an image processing 
system 20 as an image processing means, a 
recording/reproduction system 30, and a control system 
40. Furthermore, the image sensing system 10 includes 
a photographing lens 100, a stop 110, and the 
solid-state image sensing element 120. The image 
processing system 20 includes an A/D converter 500, RGB 
image processing circuit 210, and YC processing circuit 
230. The recording/reproduction system 30 includes a 
recording processing circuit 300 and reproduction 
processing circuit 310. The control system 40 includes 
a system controller 400, operation detector or 
detection circuit 430, temperature sensor 165, and 
solid-state image sensing element driving circuit 420. 
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The image sensing system 10 is an optical 
processing system for forming an optical image coming 
from an object on the image sensing surface of the 
solid-state image sensing element 120 via the stop 110 
and photographing lens 100, and exposes the solid-state 
image sensing element 120 to an object image. As 
described above, the solid-state image sensing element 
120 effectively uses an image sensing device such as a 
CCD, CMOS sensor, or the like, and an image signal that 
represents a moving image sequence or an image signal 
that represents a still image by one-shot exposure can 
be obtained by controlling the exposure time and 
interval of the solid-state image sensing element 120. 

As described above, the solid-state image sensing 
element 120 is an image sensing device which has 800 
pixels in the longitudinal direction X 600 pixels in 
the widthwise direction of each image sensing area, 
i.e., a total of 1.92 million pixels. Optical filters 
of three primary colors, i.e., red (R) , green (G) , and 
blue (B) are arranged on the front surface of the image 
sensing element 120 for respective areas. 

Image signals read out from the solid-state image 
sensing element 120 are supplied to the image 
processing system 20 via the A/D converter 500. The 
A/D converter 500 is a signal conversion circuit for 
converting an input signal into, e.g., a 10-bit digital 
signal in accordance with the amplitude of a signal of 
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each exposed pixel, and outputting the digital signal, 
and the subsequent image signal processes are 
implemented by digital processes. 

The image processing system 20 is a signal 
processing circuit for obtaining an image signal of a 
desired format from R, G, and B digital signals, and 
converts R, G, and B color signals into YC signals 
including a luminance signal Y and color difference 
signals (R-Y) and (B-Y) , or the like. 

The RGB image processing circuit 210 is a signal 
processing circuit for processing image signals of 800 
x 600 x 4 pixels received from the solid-state image 
sensing element 120 via the A/D converter 500, and has 
a white balance circuit, gamma correction circuit, and 
interpolation arithmetic circuit for achieving high 
resolution by interpolation. 

The YC processing circuit 230 is a signal 
processing circuit for generating a luminance signal Y 
and color difference signals R-Y and B-Y. The circuit 
230 comprises a high-frequency luminance generation 
circuit for generating a high-frequency luminance 
signal YH, a low-frequency luminance signal generation 
circuit for generating a low-frequency luminance signal 
YL, and a color difference signal generation circuit 
for generating color difference signals R-Y and B-Y. 
The luminance signal Y is formed by mixing the 
high- and low-frequency luminance signals YH and YL. 



The recording/reproduction system 30 is a 
processing system for outputting an image signal to a 
memory, and outputting an image signal to a liquid 
crystal monitor 4. The system 30 includes the 
5 recording processing circuit 300 for executing 

read/write processes of an image signal from/to a 
memory, and the reproduction processing circuit 310 for 
reproducing an image signal read out from the memory, 
and outputting the reproduced signal to the monitor. 
10 More specifically, the recording processing circuit 300 
includes a compression/expansion circuit for 
compressing YC signals, which represent a still image 
and moving image, in a predetermined compression format, 
and expanding compressed data when it is read out. 
15 The compression/expansion circuit has a frame 

memory and the like for signal processes, stores YC 
signals from the image processing system 20 in this 
frame memory for respective frames, reads out the 
stored signals for a plurality of blocks, and encodes 
20 them by compression coding. Compression coding is 

implemented by performing two-dimensional orthogonal 
transformation, normalization, and Huffman coding of 
image signals for respective blocks. 

The reproduction processing circuit 310 is a 
25 circuit for converting the luminance signal Y and color 
difference signals R-Y and B-Y using a matrix into, 
e.g., R, G, and B signals. The signals converted by 
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the reproduction processing circuit 310 are output to 
the liquid crystal monitor 4, thus displaying and 
reproducing a visible image. 

The control system 40 includes control circuits 
of respective units for controlling the image sensing 
system 10, image processing system 20, and 
recording/reproduction system 30 in response to an 
external operation, and controls driving of the 
solid-state image sensing element 120, the operation of 
the RGB image processing circuit 210, the compression 
process of the recording processing circuit 300, and 
the like upon detection of pressing of the release 
button 106. More specifically, the control system 40 
includes the operation detector 430 for detecting the 
operation of the release button 106, the system 
controller 400 for controlling respective units in 
response to a detection signal from the detector, and 
generating and outputting timing signals and the like 
upon sensing an image, and the solid-state image 
sensing element driving circuit 420 for generating a 
driving signal used to drive the solid-state image 
sensing element 120 under the control of the system 
controller 400. 

The RGB image processing circuit 210 executes the 
following processes. The white balance circuit in the 
RGB image processing circuit 210 executes predetermined 
white balance correction of R, G, and B signals output 



via the A/D converter 500 in correspondence with R, G, 
and B regions, and the gamma correction circuit 
executes predetermined gamma correction for these 
signals. The interpolation arithmetic circuit in the 
RGB image processing circuit 210 executes an 
interpolation process and distortion correction of 
image signals from the solid-state image sensing 
element 120 to generate R, G, and B image signals each 
having a resolution of 1200 x 1600, and supplies these 
signals to the subsequent high- and low-frequency 
luminance signal generation circuits and color 
difference signal generation circuit. 

This interpolation process includes a first 
interpolation process for correcting misregistration 
resulting from relative shifts of object images caused 
by expansion/shrinkage of the photographing lens due to 
a change in temperature or from manufacturing errors of 
the photographing lens, and a second interpolation 
process for forming a composite image signal by setting 
R, G, and B image signals to have an identical 
resolution. 

The subsequent distortion correction is an 
arithmetic process for correcting distortion of the 
photographing optical system by a known scheme. In 
this case, since R, G, and B object images have 
identical magnifications and distortions depending on 
setups of the photographing lens 100, they can undergo 



unique distortion correction. By correcting the 
distortion of the photographing optical system by 
arithmetic operations, the arrangement of the 
photographing lens 100 can be optimized to correct 
5 other optical aberrations. 

Details of the first interpolation process are as 
follows . 

Both the object image spacing and image sensing 
area spacing vary due to changes in the image sensing 

10 system 10 resulting from a change in temperature. Let 
a s be the linear expansion coefficient of the 
solid-state image sensing element 120, oc L be that of 
the photographing lens 100, AT be the temperature 
change amount, and d 0 be the spacing between 

15 neighboring lens portions. Then, since the image 
sensing system 10 has a very small imaging 
magnification, a registration change amount AZ T between 
the image sensing areas 820a and 820d can be expressed 
as the difference between expansions of the 

20 photographing lens and solid-state image sensing 
element by: 

AZ T = d 0 X (0C L - 0t s ) x AT ... (1) 

If cc s = 0.26 x 10~ 5 , AT = 20°, d 0 = 1.9832 [mm] as 
the spacing between the lens portions which form two G 
25 object images, and a L = 1.2 x 10" 5 assuming that the 

photographing lens 100 is made up of low-melting point 
glass, we have AZ T = 0.00037 [mm]. This value is the 
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spacing change amount between the two G object images, 
and also that between R and B object images. The 
temperature change amount AT is obtained by the 
temperature sensor 165. 

Fig. 19 illustrates this state, i.e., a state in 
which the positions of the same elements as those 
illustrated in Fig. 13 have changed due to thermal 
expansion. Fig. 19 is depicted with reference to the 
imaging system 855a for the sake of simplicity. The 
imaging system 855a corresponds to the lens portion 
800a in the photographing lens 800 shown in Fig. 7, and 
the imaging system 855d corresponds to the lens portion 
800d. 

Misregistration due to shifts of object images 
caused by expansion or shrinkage of the photographing 
lens is produced in a direction to connect the optical 
axes of the two imaging systems. Upon examining the 
imaging systems 855a and 855d, misregistration between 
these systems takes place in a direction parallel to 
the plane of paper of Fig. 19. The reason why this 
interpolation process is done before distortion 
correction is that the direction of misregistration has 
deviated from that parallel to the plane of paper after 
distortion correction, and interpolation cannot be 
implemented by simple arithmetic operations. 

If a temperature difference AT [°C] has occurred 
compared to that upon manufacture, the optical axis 



distance d 0 between the imaging systems 855a and 855d 
changes by d 0 X a L x AT. Since the image sensing system 
10 has a very small imaging magnification, the shift 
amount of an object image on the image sensing area 
5 856d can also be considered as d 0 x a L x AT. On the 
other hand, the solid-state image sensing element 120 
also suffers slight thermal deformation. By utilizing 
the fact that the imaging magnification is very small, 
that change amount can be expressed by d 0 x oc s x AT. 
10 Therefore, the registration change amount AZ T is 

expressed by their difference, and is given by d 0 x (CC L 
- a s ) x AT, as described above. 
If 

K T = d 0 X (cc L - 0C S ) ... (2) 

15 the registration change amount AZ T is given as the 

product of a constant and the temperature difference 
by: 

AZ T = K T X AT . • - { 3 ) 

where K T is the registration temperature coefficient of 
20 a G image. If one unit is expressed by one pixel pitch 
(pxl) , the registration change amount is given by AZ T /P 
using the pixel pitch P. 

Misregistration due to a change in temperature 
occurs among all the imaging systems. 
25 On the other hand, since misregistration due to 

manufacturing errors of the photographing lens can be 
reduced to a relatively small value, only components in 
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the same direction as misregistration due to expansion 
or shrinkage of the photographing lens will be 
considered for the sake of simplicity. 

Misregistration due to manufacturing errors of 
5 the photographing lens are defined as follows with 
reference to the positional relationship between the 
image sensing area 820a and an object image formed in 
the image circle 851a: 

Ar (pxl) : a manufacturing error of the positional 
10 relationship between the image sensing area 820b and an 
object image formed in the image circle 851b; 

Ab (pxl) : a manufacturing error of the positional 
relationship between the image sensing area 820c and an 
object image formed in the image circle 851c; 
15 Ag (pxl) : a manufacturing error of the positional 

relationship between the image sensing area 820d and an 
object image formed in the image circle 851d. 
One pixel pitch (pxl) is one unit. 
An actual first interpolation process for 
20 correcting misregistration due to a change in 

temperature and manufacturing errors mentioned above is 
done as follows. Let Gl(i,j), R(i,j), B(i,j), and 
G2(i,j) be image signals from the image sensing areas 
820a, 820b, 820c, and 820d, and addresses are defined, 
25 as shown in Fig. 20. 

Interpolated image signals G2 T (i,j), R T (i,j), and 
B T (i,j) of G2(i,j), R(i,j), and B(i,j) are generated 
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with reference to the image signal Gl(i,j) of the 
imaging system 855a (lens portion 800a) on the basis o 
equations (4) to (9) below. Equations (4) to (9) are 
used to generate by linear interpolation pixel outputs 
at imaginary positions from neighboring pixel outputs 
which are present actually. Arithmetic equations are 
selectively used depending on the signs of AZ T /P + Ag, 
AZ T /(P x V2) + Ar, and AZ T /(P x V2) + Ab. 

If the image sensing areas 820a and 820d have 
horizontal pitch a = P, vertical pitch b = P, constant 
c = 900, and positive integer h = 1 on the 
light-receiving surface, these areas have a positional 
relationship in which they are separated a x h x c in 
the horizontal direction and b x c in the vertical 
direction within the light-receiving surface. Hence, 
pixels always form lines in the direction in which 
misregistration occurs, and misregistration caused by a 
change in temperature can be corrected by very simple 
arithmetic operations. 

Note that the registration temperature 
coefficients of R and B images are 1/V2 of the 
registration temperature coefficient K T between G 
images in association with the ratio between the 
optical axis distances of the imaging systems. That is 
the distance from the lens portion 800a to the lens 
portion 800b or 800c is 1/V2 of the distance from the 
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lens portion 800a to the lens portion 800d and, as 
result, misregistration also becomes 1/V2. 
Generate G2 T (i,j) 

(1) when AZ T /P + Ag < 0 

G2 T (i,j) - (1 - (AZ T /P + Ag)) X G2(i,j) 
+ (AZ T /P + Ag) x G2(i-l,j+l) 

(2) when AZ T /P + Ag > 0 

G2 T (i,j) = (1 - (AZ T /P + Ag)) x G2(i,j) 
+ (AZ T /P + Ag) x G2(i+l,j-l) 
Generate R T (i,j) 

(3) when AZ T /(P x V2) + Ar < 0 

R T (i,j) = (1 - (AZ T /(P x V2) + Ar) ) x R(i,j) 
+ (AZ T /(P X V2) + Ar) X R(i,j+1) 

(4) when AZ T /(P X ^2) + Ar > 0 

R T (i,j) = (1 - (AZ T /(P x V2) + Ar)) x R(i,j) 
+ (AZ T /(P x V2) + Ar) X R(i,j-1) 

Generate B T (i,j) 

(5) when AZ T /(P x V2> + Ab < 0 

B T (i,j) = (1 - (AZ T /(P X V2) + Ag)) X B(i,j) 
+ (AZ T /(P x V2) + Ag) x B(i+l,j) 

(6) when AZ T /(P x V2) + Ab > 0 

B T (i,j) = (1 - (AZ T /(P x V2) + Ag) ) x B(i,j) 
+ (AZ T /(P x a/2) + Ag) x B(i,j-1) 
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The interpolated image signals G2 T (i,j), R T (i,j), 
and B T (i,j) obtained by the aforementioned processes 
are used in the second correction process. 

The second interpolation process generates G, R, 
and B image signals G' (m,n), R'(m,n), and B'(m,n) each 
having a resolution of 1200 X 1600 pixels from the 
image signal Gl(i,j) and interpolated image signals 
G2 T (i,j), R T (i,j), and B T (i,j) each having a resolution 
of 600 x 800 pixels. Equations (10) to (21) below 
describe arithmetic operations for generating pixel 
outputs at dataless positions by averaging neighboring 
pixel outputs. 
Generate G' (m, n) 

(1) when m: even number and n: odd number 

G'(m,n) = G2 T (m/2, (n+l)/2) ...(10) 

(2) when m: odd number and n: even number 

G'(m,n) = Gl ( (m+l)/2,n/2) ...(11) 

(3) when m: even number and n: even number 
G' (m,n) = (Gl(m/2,n/2) + Gl (m/2+1, B/2 ) 

+ G2 T (m/2,n/2) + G2 T (m/2 , n/2 + 1 ) ) /4 

. . . (12) 

(4) when m: odd number and n: odd number 
G'(m,n) = (Gl ( (m+1) /2, (n-1) /2) 

+ Gl ( (m+1) /2, (n-1) /2+1) 
+ G2 T ( (m-1) /2, (n+1) /2) 

+ G2 T ( (m-1) /2+1, (n+1) /2) ) /4 ...(13) 
Generate R ' (rn, n) 



(5) when m: even number and n: odd number 

R' (m,n) = (R T {m/2, (n+1) /2) + R T (m/2+1, (n+1) /2) /2 

• • • (14) 

(6) when m: odd number and n: even number 

R 1 (m, n ) = (R T ( (m+1) /2,n/2) + R T ( (m+1 ) /2 , n/2 + 1 ) /2 

. . . (15) 

(7) when m: even number and n: even number 
R'(m,n) = (R T (m/2,n/2) + R T (m/2 + 1, n/2) 

+ R T (m/2, n/2+1) + R T (m/2+1, n/2 + 1) ) /4 

. . . (16) 

(8) when m: odd number and n: odd number 

R'(m,n) = R T ( (m+1) /2, (n+1) /2) ..-(17) 
Generate B ' (m, n) 

(9) when m: even number and n: odd number 

B'(m,n) = (B T (m/2, (n-1) /2) + B T (m/2 , (n-1 ) /2 + 1 ) ) /2 

. . . (18) 

(10) when m: odd number and n: even number 

B'(m,n) = (B T ( (m-1) /2,n/2) + B T ( (m-1) /2 + 1, n/2) ) /2 

• • • (19) 

(11) when m: even number and n: even number 

B ' (m, n ) = B T (m/2,n/2) ...(20) 

(12) when m: odd number and n: odd number 
R'(m,n) - (R T (m/2, n/2) + R T (m/2 + 1, n/2) 

+ R T (m/2, n/2 + 1) + R T (m/2 + 1 , n/2 + 1 )) /4 

- - - (21) 
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As described above, after the first interpolation 
process executes position correction of at least one of 
the output images of the plurality of image sensing 
areas, the second interpolation process forms a 
composite image signal on the basis of the output 
images of the plurality of image sensing areas. 

The subsequent luminance and color difference 
signal processes using G'(m,n), R'(m,n), and B'(m,n) 
are the same as those in a normal digital color camera. 

The operation of the camera will be described 
below. Upon photographing, the camera is used while 
attaching the contact protection cap 200 that protects 
the connection terminal 114 of the camera main body 101. 
When the contact protection cap 200 is attached to the 
camera main body 101, it serves as a camera grip, and 
allows the user to easily hold the camera. 

When the main switch 105 is turned on, a power 
supply voltage is supplied to the respective units to 
allow their operation. Subsequently, it is checked if 
an image signal can be recorded on a memory. In this 
case, a photographable count is displayed on the 
indicator 150 in accordance with the remaining capacity 
of the memory, if photographing is possible based on 
the display, the operator directs the camera to an 
object field and presses the release button 106. 

When the operator has pressed the release button 
106 to its half-stroke position, the first-stage 
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circuit of the switch 121 is closed to calculate an 
exposure time. Upon completion of all photographing 
preparation processes, the camera is ready to 
photograph, and informs the photographer of that 
message. When the photographer has pressed the release 
button 106 to its full-stroke position, the 
second-stage circuit of the switch 121 is closed, and 
an operation detection circuit (not shown) sends that 
detection signal to the system controller. In this 
case, an elapse of the exposure time calculated in 
advance is counted, and after an elapse of the 
predetermined exposure time, a timing signal is 
supplied to the solid-state image sensing element 
driving circuit. In response to this signal, the 
solid-state image sensing element driving circuit 
generates horizontal and vertical driving signals to 
seguentially read out 800 x 600 exposed pixels of all 
the image sensing areas in the horizontal and vertical 
directions . 

In this case, the photographer holds the contact 
protection cap 200, and presses the release button 106 
while pinching the camera main body 101 with the index 
and thumb fingers of the right hand. Since a 
projection 106a is formed integrally with the release 
button 106 on a central line L2 of the axis of the 
release button 106, and the projection 120 is formed on 
the back cover 125 at an extended position of the 
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central line L2, the photographer makes release 
operation, i.e., push the projections 106a and 120 by 
feeling around them by the index and thumb fingers, 
respectively. In this way, generation of a couple 129 
shown in Fig. 3 can be easily prevented, and a 
high-quality image free from any blurring can be sensed. 

The readout pixels are converted into digital 
signals of a predetermined bit value by the A/D 
converter 500, and the digital signals are sequentially 
supplied to the RGB image processing circuit 210 of the 
image processing system 20. The RGB image processing 
circuit 210 executes the interpolation processes of 
these pixels after white balance adjustment and gamma 
correction, and supplies the processed signals to the 
YC processing circuit 230. 

In the YC processing circuit 230, the high- and 
low-frequency luminance signal generation circuits 
respectively compute high- and low-frequency luminance 
signals YH and YL of the R, G, and B pixels. Each 
high-frequency luminance signal YH as the computation 
result is output to an adder via a low-pass filter. 
Likewise, each low-frequency luminance signal YL is 
output to the adder via the low-pass filter after the 
high-frequency luminance signal YH is subtracted 
therefrom. In this manner, the high-frequency 
luminance signal YH and its difference YL-YH from the 
low-frequency luminance signal are added to obtain a 
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luminance signal Y. Also, the color difference signal 
generation circuit computes and outputs color 
difference signals R-Y and B-Y. The output color 
difference signals R-Y and B-Y are filtered by low-pass 
filters, and are then supplied to the recording 
processing circuit 300. 

Upon receiving the YC signals, the recording 
processing circuit 300 compresses each luminance signal 
Y and color difference signals R-Y and B-Y by a 
predetermined still image compression scheme, and 
records them on a memory. 

Upon reproducing an image from an image signal 
which is recorded on the memory and represents a still 
image or moving image, when the operator has pressed a 
reproduction button 9, the operation detection circuit 
430 detects that operation, and supplies a detection 
signal to the system controller 400. In this manner, 
the recording processing circuit 300 is driven. The 
driven recording processing circuit 300 reads out the 
recorded contents from the memory and displays an image 
on the liquid crystal monitor 4. The operator selects 
a desired image by pressing, e.g., a select button. 
(Second Embodiment) 

In the first embodiment, misregistration caused 
by changes in the image sensing system 10 (890) due to 
a change in temperature is corrected based on the 
output from the temperature sensor 165. 
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Misregistration also occurs due to a change in object 
distance. The second embodiment corrects such 
misregistration caused by a change in object distance. 

Fig. 21 shows a signal processing system which 
uses an output from a distance measurement device to 
correct misregistration. The same reference numerals 
denote the same components as those in the first 
embodiment . 

As shown in Fig. 21, an image sensing apparatus 
has an image sensing system 10, an image processing 
system 20 as image processing means, a 

recording/reproduction system 30, and a control system 
40. Furthermore, the image sensing system 10 includes 
a photographing lens 100, stop 110, and solid-state 
image sensing element 120. The image processing system 
20 includes an A/D converter 500, RGB image processing 
circuit 410, and YC processing circuit 230. The 
recording/reproduction system 30 includes a recording 
processing circuit 300 and reproduction processing 
circuit 310. The control system 4 0 includes a system 
controller 400, operation detector or detection circuit 
430, distance measurement device 465, and solid-state 
image sensing element driving circuit 420. 

The distance measurement device 4 65 can adopt an 
active distance measurement device which outputs 
distance by projecting ultrasonic waves or LED light to 
an object, or a passive distance measurement device 
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which outputs distance based on the principle of 
triangulation using an object image. 

The image sensing system 10 is an optical 
processing system for forming an optical image coming 
from an object on the image sensing surface of the 
solid-state image sensing element 120 via the stop 110 
and photographing lens 100, and exposes the solid-state 
image sensing element 120 to an object image. As 
described above, the solid-state image sensing element 
120 effectively uses an image sensing device such as a 
CCD, CMOS sensor, or the like, and an image signal that 
represents a moving image sequence or an image signal 
that represents a still image by one-shot exposure can 
be obtained by controlling the exposure time and 
interval of the solid-state image sensing element 120. 

As in the first embodiment, the solid-state image 
sensing element 120 is an image sensing device which 
has 800 pixels in the longitudinal direction x 600 
pixels in the widthwise direction of each image sensing 
area, i.e., a total of 1.92 million pixels. Optical 
filters of three primary colors, i.e., red (R) , green 
(G) , and blue (B) are arranged on the front surface of 
the image sensing element 120 for respective areas. 
Both the vertical and horizontal pixel pitches are 1.56 
Jim. 

As shown in Fig. 21, image signals read out from 
the solid-state image sensing element 120 are supplied 



to the image processing system 20 via the A/D converter 
500. The A/D converter 500 is a signal conversion 
circuit for converting an input signal into, e.g., a 
10-bit digital signal in accordance with the amplitude 
of a signal of each exposed pixel, and outputting the 
digital signal, and the subsequent image signal 
processes are implemented by digital processes. 

The image processing system 20 is a signal 
processing circuit for obtaining an image signal of a 
desired format from R, G, and B digital signals, and 
converts R, G, and B color signals into YC signals 
including a luminance signal Y and color difference 
signals (R-Y) and (B-Y) , or the like. 

The RGB image processing circuit 410 is a signal 
processing circuit for processing image signals of 800 
X 600 X 4 pixels received from the solid-state image 
sensing element 120 via the A/D converter 500, and has 
a white balance circuit, gamma correction circuit, and 
interpolation arithmetic circuit for achieving high 
resolution by interpolation. 

The YC processing circuit 230 is a signal 
processing circuit for generating a luminance signal Y 
and color difference signals R-Y and B-Y. The circuit 
230 comprises a high-frequency luminance generation 
circuit for generating a high-frequency luminance 
signal YH, a low-frequency luminance signal generation 
circuit for generating a low-frequency luminance signal 



YL, and a color difference signal generation circuit 
for generating color difference signals R-Y and B-Y. 
The luminance signal Y is formed by mixing the 
high- and low-frequency luminance signals YH and YL. 

The recording/reproduction system 30 is a 
processing system for outputting an image signal to a 
memory, and outputting an image signal to a liquid 
crystal monitor 4. The system 30 includes the 
recording processing circuit 300 for executing 
read/write processes of an image signal from/to a 
memory, and the reproduction processing circuit 310 for 
reproducing an image signal read out from the memory, 
and outputting the reproduced signal to the monitor. 
More specifically, the recording processing circuit 300 
includes a compression/expansion circuit for 
compressing YC signals, which represent a still image 
and moving image, in a predetermined compression format, 
and expanding compressed data when it is read out. 

The compression/expansion circuit has a frame 
memory and the like for signal processes, stores YC 
signals from the image processing system 20 in this 
frame memory for respective frames, reads out the 
stored signals for a plurality of blocks, and encodes 
them by compression coding. Compression coding is 
implemented by performing two-dimensional orthogonal 
transformation, normalization, and Huffman coding of 
image signals for respective blocks. 



The reproduction processing circuit 310 is a 
circuit for converting the luminance signal Y and color 
difference signals R-Y and B-Y using a matrix into, 
e.g., R, G, and B signals. The signals converted by 
the reproduction processing circuit 310 are output to 
the liquid crystal monitor 4, thus displaying and 
reproducing a visible image. 

The control system 40 includes control circuits 
of respective units for controlling the image sensing 
system 10, image processing system 20, and 
recording/reproduction system 30 in response to an 
external operation, and controls driving of the 
solid-state image sensing element 120, the operation of 
the RGB image processing circuit 410, the compression 
process of the recording processing circuit 300, and 
the like upon detection of pressing of a release button 
6. More specifically, the control system 40 includes 
the operation detector 430 for detecting the operation 
of the release button 6, the system controller 400 for 
controlling respective units in response to a detection 
signal from the detector, and generating and outputting 
timing signals and the like upon sensing an image, and 
the solid-state image sensing element driving circuit 
420 for generating a driving signal used to drive the 
solid-state image sensing element 120 under the control 
of the system controller 400. 



The RGB image processing circuit 410 executes the 
following processes. The white balance circuit in the 
RGB image processing circuit 410 executes predetermined 
white balance correction of R, G, and B signals output 
via the A/D converter 500 in correspondence with R, G, 
and B regions, and the gamma correction circuit 
executes predetermined gamma correction for these 
signals. The interpolation arithmetic circuit in the 
RGB image processing circuit 410 executes an 
interpolation process and distortion correction of 
image signals from the solid-state image sensing 
element 120 to generate R, G, and B image signals each 
having a resolution of 1200 X 1600, and supplies these 
signals to the subsequent high- and low-frequency 
luminance signal generation circuits and color 
difference signal generation circuit. 

This interpolation process includes a first 
interpolation process for correcting misregistration 
caused by a change in object distance, and a second 
interpolation process for forming a composite image 
signal by setting R, G, and B image signals to have an 
identical resolution. 

The subsequent distortion correction is an 
arithmetic process for correcting distortion of the 
photographing optical system by a known scheme. In 
this case, since R, G, and B object images have 
identical magnifications and distortions depending on 
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setups of the photographing lens 100, they can undergo 
unique distortion correction. By correcting the 
distortion of the photographing optical system by 
arithmetic operations, the arrangement of the 
5 photographing lens 100 can be optimized to correct 
other optical aberrations. 

Details of the first interpolation process are as 
follows. In the following description, an object 
distance will be examined with reference to the imaging 

10 system for the sake of simplicity. A registration 

change amount AZ S between the image sensing areas 820a 
and 820b can be expressed as a function of a difference 
AS between an actual object distance and reference 
object distance St by: 

15 AZ S = d 0 -S'-AS/{St- (AS + St)} ...(22) 

The difference AS is obtained from the output of the 
distance measurement device 4 65. 

Assuming that arrangement of the image sensing 
system is the same as that in the first embodiment, the 

20 spacing d 0 of lens portions = 1.9832 [mm], the 

reference object distance St = 2380 - 1.45 = 2378.55 
[mm] , and the spacing S ' between the imaging system and 
image sensing area = 1.4 5 [mm]. 

Fig. 22 shows an optical path when the actual 

25 object distance increases to infinity. Each image 

sensing area has a size of 1.248 mm x 0.936 mm, and the 
image sensing areas neighbor in the horizontal 
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direction via a 0.156 mm (horizontal) x 0.468 mm 
(vertical) separation band. The spacing between the 
centers of the neighboring image sensing areas is 1.404 
mm in the vertical and horizontal directions, and is 
1.9856 mm in the diagonal direction. 

Paying attention to the image sensing areas 820a 
and 820d, assume that an image of an object located at 
a position of a reference object distance is formed on 
the image sensing portion at intervals of 1.9845 mm 
obtained by subtracting the diagonal size for a 0.5 
pixel from 1.9856 mm as the image sensing area spacing 
so as to implement pixel shift. As a result, the 
spacing between the lens portions 800a and 800d of the 
photographing lens 800 is set at 1.9832 mm, as shown in 
Fig. 22. In Fig. 22, arrows 855a and 855d are symbols 
that represent imaging systems having a positive power 
by the lens portions 800a and 800d of the photographing 
lens 800, rectangles 856a and 856d are symbols that 
represent the ranges of the image sensing areas 820a 
and 820d, and L801 and L802 indicate optical axes of 
the imaging systems 855a and 855d. 

The difference AS between the actual object 
distance and reference object distance St is infinity. 
When AS °o, the limit of equation (22) is: 




d 0 -S' _ 1,9832x1.45 
St 2378.55 



= 0.0012[mm] 



. . . (23) 
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As can be seen from this equation, misregistration of 
0.0012 [mm] is produced. If one unit is expressed by 
one pixel pitch (pxl), the registration change amount 
is given by AZ S /P using the pixel pitch P. 
5 An actual first interpolation process for 

correcting misregistration due to a change in object 
distance mentioned above is done as follows. As in the 
first embodiment, let Gl(i,j), R(i,j), B(i,j), and 
G2(i,j) be image signals from the image sensing areas 

10 820a, 820b, 820c, and 820d, and addresses are defined, 
as shown in Fig. 20. 

Interpolated image signals G2 s (i,j), R s (i,j), and 
B s (i,j) of G2(i,j), R(i,j), and B(i,j) are generated 
with reference to the image signal Gl(i,j) of the 

15 imaging system 855a (lens portion 800a) on the basis of 
equations (24) to (29) below. Equations (24) to (29) 
are used to generate pixel outputs at imaginary 
positions from neighboring pixel outputs which are 
present actually by linear interpolation. If P 

20 represents the pixel pitch, and the image sensing areas 
820a and 820d have horizontal pitch a = P, vertical 
pitch b = P, constant c = 900, and positive integer h = 
1 on the light-receiving surface, these areas have a 
positional relationship in which they are separated a x 

25 h x c in the horizontal direction and b x c in the 

vertical direction within the light-receiving surface. 
Hence, misregistration caused by a change in 



temperature or a change in object distance can be 
corrected by very simple arithmetic operations. 

Note that misregistration between R and B images 
are 1/V2 of that between G images in association with 
the ratio between the optical axis distances of the 
imaging systems. 
Generate G2 s (i,j) 

(1) when AZ S > 0 

G2 s (i,j) = (1 - AZ S /P) X G2(i,j) + (AZs/P) 

X G2(i-1, j+1) . . . (24) 

(2) when AZ S < 0 

G2 s (i,j) = (1 - AZ S /P) x G2(i,j) + (Z s /P) 

x G2 (i+1, j-1) . . . (25) 

Generate R s (i,j) 

(3) when AZ S > 0 

Rs(i/j) = (1 - AZ S /(P x V2)) x R(i,j) 

+ (AZ s /(PxV2)) x R(i,j+1) ...(26) 

(4) when AZ S < 0 

Rs(i,j) = (1 - AZ S /(P x V2)) x R(i,j) 

+ (AZ s /(PxV2)) x R(i,j-1) ...(27) 
Generate B s (i,j) 

(3) when AZ S > 0 

B s (i,j) = (1 - AZ S /(P X a/2)) x B(i,j) 

+ (AZ s /(PxV2)) x B(i+l,j) ...(28) 

(4) when AZ S < 0 

B s (i,j) = (1 - (AZ S /(P X V2)) X B(i,j) 

+ (AZ S / (P x V2) ) X B(i,j-1) ...(29) 
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The interpolated image signals G2 s (i,j), R g (i,j), 
and B s (i,j) obtained by the aforementioned processes 
are free from any misregistration caused by a change in 
object distance, and are then used in the second 
correction process. 
(Third Embodiment) 

In the second embodiment, misregistration caused 
by a change in object distance is corrected on the 
basis of the output from the distance measurement 
device 4 65. In the third embodiment, such 
misregistration is corrected without using the output 
from the distance measurement device 4 65. 

A switch 109 (Fig. 2) which is used by the user 
to set up the state of the camera is used as a macro 
photographing mode setting switch. When a 
near-distance object is to be photographed as the next 
object, the photographer presses the macro 
photographing mode setting switch 109. Upon detection 
of pressing of the macro photographing mode setting 
switch 109 by the operation detection circuit 430, for 
example, an interpolation process is done while the 
difference AS from the reference object distance St is 
set at -1.19 [m] . That is, AZ S is calculated while AS 
= -1.19 in eguation (22), and an interpolation process 
is executed based on eguations (24) to (29). 
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With this arrangement, since no distance 
measurement device is required, a cost reduction can be 
attained. 

In addition, it is advantageous to achieve a 
5 compact, low-profile camera. 
(Fourth Embodiment) 

Misregistration caused by a change in temperature 
is corrected based on the output from the temperature 
sensor 165 in the first embodiment, and that caused by 
10 a change in object distance is corrected based on the 
output from the distance measurement device 4 65 in the 
second embodiment. In the fourth embodiment, 
misregistration is corrected based on the output itself 
from the solid-state image sensing element without 
15 using the outputs from the temperature sensor 165 and 
distance measurement device 4 65. At this time, 
deviation of images caused by disparity is detected by 
exploiting correlation in an oblique direction between 
two G image sensing areas. With this arrangement, 
20 misregistration caused by a change in temperature, that 
caused by a change in object distance, and that caused 
by manufacturing errors of the photographing lens can 
be corrected together. 

Fig. 23 is a view for explaining pixels on the 
25 solid-state image sensing element, which are used to 
detect misregistration. Fig. 23 shows the positional 
relationship between the object images and image 
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sensing areas while reducing the numbers of pixels to 
1/100 in the vertical and horizontal directions. 

Referring to Fig. 23, reference numerals 320a, 
320b, 320c, and 320d denote four image sensing areas of 
the solid-state image sensing element 820. For the 
sake of simplicity, each of the image sensing areas 
320a, 320b, 320c, and 320d has an 8 X 6 pixel matrix. 
The image sensing areas 320a and 320d, 320b, and 320c 
respectively output G, R, and B image signals. Pixels 
in the image sensing areas 320a and 320d are indicated 
by open rectangles, those in the image sensing area 
320b are indicated by hatched rectangles, and those in 
the image sensing area 320c are indicated by closed 
rectangles . 

A separation band having a size of one pixel 
(horizontal) x 3 pixels (vertical) is formed between 
neighboring image sensing areas. Therefore, the 
distances between the centers of the image sensing 
areas for outputting G images are egual to each other 
in the horizontal and vertical directions. 

Reference numerals 351a, 351b, 351c, and 351d 
denote object images. To attain a pixel shift, centers 
360a, 360b, 360c, and 360d of the object images 351a, 
351b, 351c, and 351d are offset 1/4 pixels from the 
centers of the image sensing areas 320a, 320b, 320c, 
and 320d in the direction of a center 320e of the 
entire image sensing area. 
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As a result, upon projecting the image sensing 
areas backward onto a plane at a predetermined distance 
on the object field side, backward projected images of 
the centers 360a, 360b, 360c, and 360d of the object 
images overlap each other, and those of the pixels of 
the image sensing areas 320a, 320b, 320c, and 320d are 
laid out in a mosaic pattern so that their centers do 
not overlap each other. 

Reference numerals L301 and L302 denote line 
segments which represent pixel sequences used to detect 
misregistration. Since both the image sensing areas 
320a and 320d output G images, signals from a pixel 
sequence which are located within the image sensing 
area 320a and under the line segment L301 form a shape 
similar to that formed by signals from a pixel sequence 
which are located within the image sensing area 320d 
and under the line segment L301. When these pixel 
signals undergo correlation arithmetic operations, 
misregistration between the image sensing areas 320a 
and 320d can be detected. Likewise, by making 
correlation arithmetic operations of signals from a 
pixel sequence which are located within the image 
sensing area 320a and under the line segment L302, and 
signals from a pixel sequence which are located within 
the image sensing area 32 Od and under the line segment 
L302, misregistration between the image sensing areas 
320a and 320d can also be detected. 



In this embodiment, misregistration amounts are 
calculated using two pairs of image signals, and the 
calculation results are averaged to improve the 
detection precision. 

Fig. 24 shows signals from a pixel sequence which 
are located within the image sensing area 320a and 
under the line segment L301, and those from a pixel 
sequence which are located within the image sensing 
area 320d and under the line segment L301. In Fig. 24, 
signals 370 indicated by black dots are those from a 
pixel sequence which are located within the image 
sensing area 320a and under the line segment L301, and 
signals 371 indicated by white dots are those from a 
pixel sequence which are located within the image 
sensing area 320d and under the line segment L301. 
Especially, reference numeral 361' denotes an output 
from a pixel 361 in Fig. 23; 362', an output from a 
pixel 362; 363', an output from a pixel 363; and 364', 
an output from a pixel 364. All these signals are 
components of a G image signal. More specifically, 
since the signals 370 and 371 have identical spectrum 
distributions in color characteristics, the correlation 
arithmetic operation result indicates an image signal 
shift with very high precision. 

Fig. 25 shows signals from a pixel sequence which 
are located within the image sensing area 320a and 
under the line segment L302, and those from a pixel 



sequence which are located within the image sensing 
area 320d and under the line segment L302. In Fig. 25, 
signals 372 indicated by black dots are those from a 
pixel sequence which are located within the image 
sensing area 320a and under the line segment L302, and 
signals 373 indicated by white dots are those from a 
pixel sequence which are located within the image 
sensing area 320d and under the line segment L302. 
Especially, reference numeral 365' denotes an output 
from a pixel 365 in Fig. 23; 366', an output from a 
pixel 366; 367' , an output from a pixel 367; and 368', 
an output from a pixel 368. All these signals are also 
components of a G image signal. That is, the signals 
372 and 373 have identical spectrum distributions in 
color characteristics. 

Figs. 24 and 25 show states which are free from 
any change in temperature, and in which an object is 
located at a position of the reference object distance. 
In these states, the signals relatively laterally 
deviate 0.5 pixels from each other as a result of pixel 
shift. 

When a change in temperature has occurred or when 
the object is separated from the position of the 
reference object distance with respect to the above 
states, signal shift takes place. For example, when 
the object is farther than the reference object 
distance position, the signals 370 (372) and 371 (373) 



respectively shift in the directions of arrows A and B 
in Fig. 24 (25) ; when the object is closer than the 
reference object distance position, they shift in 
opposite directions. This behavior is described by 
5 equation (22) as a function of the difference AS 

between the actual object distance and reference object 
distance St. 

A relative position change amount between a pair 
of signals having identical spectral distributions in 

10 color characteristics can be detected using a known 
method that adopts correlation arithmetic operations, 
e.g., using a scheme disclosed in Japanese Patent 
Publication No. 5-88445. By subtracting a design shift 
amount "0.5" from that output, the registration change 

15 amount AZ S /P that expresses one unit as one pixel pitch 
(pxl) is obtained. 

Normally, high-frequency components beyond the 
Nyquist frequency of the solid-state image sensing 
element 820 are projected onto the image sensing 

20 element 820 in terms of the characteristics of the 

optical system. Therefore, the object image phase is 
not always perfectly reflected in the signal phase 
depending on object patterns. In order to eliminate 
detection errors of the registration change amount 

25 AZ S /P resulting from such characteristics, the 

registration change amount obtained from signals from a 
pixel sequence located under the line segment L301 and 



that obtained from signals from a pixel sequence 
located under the line segment L302, which has an 
object image sampling position shifted 0.5 pixels from 
that under the line segment L301, are averaged. Since 
the obtained registration change amount has high 
precision, the first interpolation process described by 
equations (24) to (29) can be ideally done using that 
amount . 

The interpolated image signals G2 s (i,j), R s (i,j), 
and B s (i,j) obtained by the aforementioned processes 
are used in the second correction process. 

Unlike in the second embodiment, the interpolated 
image signals G2 s (i,j), R s (i,j), and B s (i,j) have 
undergone corrections of misregistration caused by a 
change in temperature, that caused by a change in 
object distance, and that caused by manufacturing 
errors of the photographing lens together in this case. 

The subsequent second interpolation process 
generates G, R, and B image signals G'(m,n), R' (m, n) , 
and B' (m,n) each having a resolution of 1200 X 1600 
pixels from the image signal Gl(i,j) and interpolated 
image signals G2 T (i,j), R T (i,j), and B T (i,j) each having 
a resolution of 600 x 800 pixels. In this embodiment, 
G2 T (i,j), R T (i,j), and B T (i,j) in equations (10) to (21) 
in the first embodiment are replaced by G2 s (i,j), 
Rs (i, j) , and B s (i, j ) . 
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As described above, the image sensing element 
outputs two G images which are formed to have identical 
spectral distributions and have substantially the same 
fields of view, and R and B images which have 
substantially the same fields of view as the two G 
images, the positions of the R and B images are 
corrected based on a change in spacing between the two 
G images in the first interpolation process, and a 
composite image signal is formed based on the R, G, and 
B images in the second interpolation process. 

If area-based matching disclosed in Japanese 
Patent Laid-Open No. 10-289316 is used, misregistration 
can be obtained for respective pixels. When 
misregistration is corrected using such scheme, even 
when an object has a depth, an image which is free from 
color misregistration over the entire frame and has 
high sharpness can be obta ined. In this case, AZs in 
equations (24) to (29) can be processed as a function 
of a pixel address. 

Fig. 26 is a view for explaining another setup of 
pixels on the solid-state image sensing element used to 
detect misregistration. 

Referring to Fig. 26, reference numerals 520a, 
520b, 520c, and 520d denote four image sensing areas of 
the solid-state image sensing element 820. For the 
sake of simplicity, each of the image sensing areas 
520a, 520b, 520c, and 520d has an 8 X 4 pixel matrix, 



but the number of pixels is increased to obtain a 
practical resolution in practice. The image sensing 
areas 520a and 520d, 520b, and 520c respectively output 
G, R, and B image signals. Pixels in the image sensing 
5 areas 520a and 520d are indicated by open rectangles, 
those in the image sensing area 520b are indicated by 
hatched rectangles, and those in the image sensing area 
520c are indicated by closed rectangles. 

A separation band having a size of one pixel 
10 (horizontal) x zero pixel (vertical) is formed between 
neighboring image sensing areas. Therefore, the 
distances between the centers of the two image sensing 
areas for outputting G images are 9 pixels in the 
horizontal direction and 4.5 pixels in the vertical 
15 direction. If the image sensing areas 820a and 820d 
have horizontal pitch a = P, vertical pitch b = P, 
constant c = 4.5, and positive integer h = 2 on the 
light-receiving surface, these areas have a positional 
relationship in which they are separated a x h X c in 
20 the horizontal direction and b x c in the vertical 
direction within the light-receiving surface. With 
this relationship, since misregistration caused by a 
change in temperature or a change in object distance 
takes place in a direction in which pixels are arranged, 
25 such misregistration can be corrected by very simple 
arithmetic operations. 
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Reference numerals 551a, 551b, 551c, and 551d 
denote object images. To attain pixel shift, centers 
560a, 560b, 560c, and 560d of the object images 551a, 
551b, 551c, and 551d are offset 1/4 pixels in the 
5 vertical and horizontal directions from the centers of 
the image sensing areas 520a, 520b, 520c, and 520d in 
the direction to approach a center 520e of the entire 
image sensing area. 

As a result, upon projecting the image sensing 
10 areas backward onto a plane at a predetermined distance 
on the object field side, backward projected images of 
the centers 560a, 560b, 560c, and 560d of the object 
images overlap each other, and those of the pixels of 
the image sensing areas 520a, 520b, 520c, and 520d are 
15 laid out in a mosaic pattern so that their centers do 
not overlap each other. 

Reference numerals L501 and L502 denote line 
segments which represent pixel sequences used to detect 
misregistration. To help easy understanding, pixels 
20 used are hatched. Since both the image sensing areas 
520a and 520d output G images, signals from a pixel 
sequence which are located within the image sensing 
area 520a and under the line segment L501 form a shape 
similar to that formed by signals from a pixel sequence 
25 which are located within the image sensing area 520d 
and under the line segment L501. When the correlation 
between these pixel signals is calculated, 



misregistration between the image sensing areas 520a 
and 520d can be detected. Likewise, by calculating the 
correlation between signals from a pixel sequence which 
are located within the image sensing area 520a and 
5 under the line segment L502, and signals from a pixel 
sequence which are located within the image sensing 
area 520d and under the line segment L502, 
misregistration between the image sensing areas 520a 
and 520d can also be detected. 
10 in this embodiment, misregistration amounts are 

calculated using two pairs of image signals, and the 
calculation results are averaged to improve the 
detection precision. 

When a change in temperature has occurred or when 
15 the object is separated from the position of the 

reference object distance with respect to the above 
states, signal shift takes place. This behavior is 
described by equation (22) as a function of the 
difference AS between the actual object distance and 
20 reference object distance St. 

A relative position change amount between a pair 
of signals having identical spectral distributions in 
color characteristics can be detected using a known 
method that adopts correlation arithmetic operations, 
25 e.g., using a scheme disclosed in Japanese Patent 

Publication No. 5-88445. By subtracting a design shift 
amount "0.5" from that output, the registration change 
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amount AZ S /P that expresses one unit as one pixel pitch 
(pxl) is obtained. Note that every other pixels must 
be used. Using the obtained registration change amount, 
the first interpolation process described by equations 
(24) to (29) can be implemented. 

Upon reversing the aspect ratio of each image 
sensing area, a separation band having a size of one 
pixel (vertical) X zero pixel (horizontal) is formed 
between neighboring image sensing areas. Therefore, 
the distances between the centers of the two image 
sensing areas for outputting G images are 9 pixels in 
the vertical direction and 4.5 pixels in the horizontal 
direction. If these two image sensing areas have 
horizontal pitch a = P, vertical pitch b = P, constant 
c = 4.5, and positive integer h = 2 on the 
light-receiving surface, these areas have a positional 
relationship in which they are separated a x c in the 
horizontal direction and b X h x c in the vertical 
direction within the light-receiving surface. 

Fig. 27 is a view for explaining still another 
setup of pixels on the solid-state image sensing 
element used to detect misregistration. 

Referring to Fig. 27, reference numerals 620a, 
620b, 620c, and 620d denote four image sensing areas of 
the solid-state image sensing element 820. For the 
sake of simplicity, each of the image sensing areas 
620a, 620b, 620c, and 620d has an 8 x 6 pixel matrix. 



The image sensing areas 620a and 620d, 620b, and 620c 
respectively output G, R, and B image signals. Pixels 
in the image sensing areas 620a and 620d are indicated 
by open rectangles, those in the image sensing area 
620b are indicated by hatched rectangles, and those in 
the image sensing area 620c are indicated by closed 
rectangles . 

A separation band having a size of one pixel 
(vertical) X three pixels (horizontal) is formed 
between neighboring image sensing areas. Therefore, 
the distances between the centers of the image sensing 
areas are egual to each other in the horizontal and 
vertical directions. 

Reference numerals 651a, 651b, 651c, and 651d 
denote object images. To attain pixel shift, centers 
660a, 660b, 660c, and 660d of the object images 651a, 
651b, 651c, and 651d are offset 1/4 pixels from the 
centers of the image sensing areas 620a, 620b, 620c, 
and 620d in the direction of a center 620e of the 
entire image sensing area. 

As a result, upon projecting the image sensing 
areas backward onto a plane at a predetermined distance 
on the object field side, backward projected images of 
the centers 660a, 660b, 660c, and 660d of the object 
images overlap each other, and those of the pixels of 
the image sensing areas 620a, 620b, 620c, and 620d are 



laid out in a mosaic pattern so that their centers do 
not overlap each other. 

Reference numerals L601 and L602 denote line 
segments which represent pixel sequences used to detect 
misregistration. Since both the image sensing areas 
620a and 620d output G images, misregistration between 
the image sensing areas 620a and 620d can be detected 
by calculating the correlation between signals from a 
pixel sequence which are located within the image 
sensing area 620a and under the line segment L601, and 
signals from a pixel sequence which are located within 
the image sensing area 620d and under the line segment 
L601. Likewise, by calculating the correlation between 
signals from a pixel sequence which are located within 
the image sensing area 620a and under the line segment 
L602, and signals from a pixel sequence which are 
located within the image sensing area 620d and under 
the line segment L602, misregistration between the 
image sensing areas 620a and 620d can also be detected. 

In this embodiment, misregistration amounts are 
calculated using two pairs of image signals, and the 
calculation results are averaged to improve the 
detection precision. 

When a change in temperature has occurred or when 
the object is separated from the position of the 
reference object distance with respect to the above 
states, signal shift takes place. This behavior is 
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described by equation (22) as a function of the 
diffe rence AS between the actual object distance and 
reference object distance St. 

A relative position change amount between a pair 
of signals having identical spectral distributions in 
color characteristics can be detected using a known 
method that adopts correlation arithmetic operations, 
e.g., using a scheme disclosed in Japanese Patent 
Publication No. 5-88445. By subtracting a design shift 
amount "0.5" from that output, the registration change 
amount AZ S /P that expresses one unit as one pixel pitch 
(pxl) is obtained. Note that every other pixels must 
be used. Using the obtained registration change amount, 
the first interpolation process described by equations 
(24) to (29) can be implemented. 
[Another Embodiment] 

Note that the present invention may be applied to 
either a system constituted by a plurality of devices 
(e.g., a host computer, interface device, reader, 
printer, and the like) , or an apparatus consisting of a 
single equipment (e.g., a copying machine, facsimile 
apparatus, or the like) . 

The objects of the present invention are also 
achieved by supplying a storage medium (or recording 
medium) , which records a program code of a software 
program that can implement the functions of the 
above-mentioned embodiments to the system or apparatus, 



and reading out and executing the program code stored 
in the storage medium by a computer (or a CPU or MPU) 
of the system or apparatus. In this case, the program 
code itself read out from the storage medium implements 
the functions of the above-mentioned embodiments, and 
the storage medium which stores the program code 
constitutes the present invention. The functions of 
the above-mentioned embodiments may be implemented not 
only by executing the readout program code by the 
computer but also by some or all of actual processing 
operations executed by an operating system (OS) running 
on the computer on the basis of an instruction of the 
program code. 

Furthermore, the functions of the above-mentioned 
embodiments may be implemented by some or all of actual 
processing operations executed by a CPU or the like 
arranged in a function extension card or a function 
extension unit, which is inserted in or connected to 
the computer, after the program code read out from the 
storage medium is written in a memory of the extension 
card or unit. 

As described above, in an image sensing apparatus 
which has a plurality of apertures for receiving 
external light from different positions, a plurality of 
image sensing means for independently receiving the 
external light received via the plurality of apertures, 
and independently extracting predetermined color 



components for each external light, and image 
processing means for processing an output signal from 
the image sensing means, the image processing means 
performs position correction of at least one of output 
images form the plurality of image sensing means, and 
then forms a composite image signal based on the output 
images from the plurality of image sensing means. 
Hence, the following effects are obtained. 

(1) A position correction method of images 
suitable for an image sensing apparatus for a compact 
digital color camera can be obtained. 

(2) As a result, a high-resolution image free 
from any color misregistration and sharpness drop can 
be easy obtained. 

Furthermore, in an image sensing apparatus which 
has an image sensing element having a plurality of 
image sensing areas, a photographing optical system for 
forming object images on the plurality of image sensing 
areas via a plurality of imaging systems corresponding 
to the plurality of image sensing areas, image 
processing means for processing an output signal from 
the image sensing element, and temperature measurement 
means, the image sensing element outputs a plurality of 
images having substantially the same fields of view, 
and the image processing means performs position 
correction of at least one of the plurality of images 
on the basis of the output from the temperature 
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measurement means, and then forms a composite image 
signal based on the plurality of images. Hence, the 
following effect can be obtained. 

(3) A method suitable for correcting 
misregistration caused by a change in temperature can 
be obtained. 

Moreover, in an image sensing apparatus which has 
an image sensing element having a plurality of image 
sensing areas, a photographing optical system for 
forming object images on the plurality of image sensing 
areas via a plurality of imaging systems corresponding 
to the plurality of image sensing areas, and image 
processing means for processing an output signal from 
the image sensing element, the image sensing element 
outputs a plurality of images having substantially the 
same fields of view, and the image processing means 
performs position correction of at least one of the 
plurality of images on the basis of object distance 
information, and then forms a composite image signal 
based on the plurality of images. Hence, the following 
effect can be obtained. 

(4) A method suitable for correcting 
misregistration caused by a change in object distance 
can be obtained. 

Also, upon setting a macro photographing mode, 
since position correction for at least one of the 
plurality of images is done, and a composite image 



signal is then formed based on the plurality of images, 
the following effects can be obtained. 

(5) A method suitable for correcting 
misregistration caused by a change in object distance 

5 upon setting the macro photographing mode can be 
obtained. 

(6) A distance measurement device can be omitted. 
In addition, in an image sensing apparatus which 

has an image sensing element having a plurality of 

10 image sensing areas on a single plane, a photographing 
optical system for forming object images on the 
plurality of image sensing areas, respectively, and 
image processing means for processing an output signal 
from the image sensing element, the image sensing 

15 element outputs first and second images which are 

formed to have an identical spectral distribution and 
have substantially the same fields of view, and a third 
image which is formed to have a spectral distribution 
different from the first and second images, and has 

20 substantially the same field of view as the first and 
second images, and the image processing means corrects 
the position of the third image on the basis of a 
change in spacing between the first and second images 
and then forms a composite image signal based on the 

25 first, second, and third images during processing of 
the output signal. Hence, the following effects are 
obtained. 
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(7) A method suitable for correcting 
misregistration based on the output itself of a 
solid-state image sensing element can be obtained. 
Hence, with this arrangement, neither a temperature 

5 sensor nor a distance measurement device are required. 

(8) This results in size and cost reductions of 
the camera. 

(9) In addition, when misregistration is 
detected for respective pixels by, e.g., area-based 

10 matching, and different misregistration correction 
schemes are adopted for respective pixels or small 
regions, a high-resolution image of even an object with 
a depth, which has various distances, can be obtained 
over the entire frame. 

15 Moreover, in an image sensing apparatus which has 

an image sensing element having first and second image 
sensing areas with nearly the same sizes on a single 
plane, a photographing optical system for respectively 
forming first and second object images on the first and 

20 second image sensing areas, and image processing means, 
each of the first and second image sensing areas has a 
matrix of a plurality of pixels arranged at a pitch a 
in the horizontal direction and a pitch b in the 
vertical direction on a light-receiving surface, the 

25 first and second image sensing areas have a positional 
relationship in which the first and second image 
sensing areas are separated a x h x c in the horizontal 
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direction and b x c in the vertical direction (where h 
is a positive integer) , the image sensing element forms 
first and second images which are formed to have an 
identical spectral distribution and have substantially 
the same fields of view, and the image processing means 
generates a composite image signal based on the first 
and second images. Hence, the following effect can be 
obtained. 

(10) A layout of image sensing areas, which 
allows to correct misregistration caused by a change in 
temperature or a change in object distance using very 
simple arithmetic operations, can be obtained. 

In addition, since a change in spacing between 
the first and second images is corrected during 
processing of the output signal, and a composite image 
signal is formed based on the first and second images, 
the following effect can be obtained. 

(11) Using the layout of image sensing areas, 
which allows to correct misregistration using very 
simple arithmetic operations, a very high-resolution 
video signal can be formed. 

As described above, according to the above 
embodiments, misregistration among R, G, and B images 
can be satisfactorily corrected, and these images can 
be composited. 

The present invention is not limited to the above 
embodiments and various changes and modifications can 



be made within the spirit and scope of the present 
invention. Therefore, to apprise the public of the 
scope of the present invention the following claims are 
made . 
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